Skip to content

feat(queue/sql): add subscriber with partition leasing#23

Merged
behinddwalls merged 1 commit into
mainfrom
sql-queue-subscriber
Feb 20, 2026
Merged

feat(queue/sql): add subscriber with partition leasing#23
behinddwalls merged 1 commit into
mainfrom
sql-queue-subscriber

Conversation

@behinddwalls
Copy link
Copy Markdown
Collaborator

@behinddwalls behinddwalls commented Feb 17, 2026

Summary

Why?

Need Subscriber implementation with partition leasing to enable distributed message processing from SQL queue.

What?

  • Subscriber discovers partitions and acquires leases for distributed work
  • Partition leasing ensures each partition processed by exactly one worker
  • Automatic failover via stale lease stealing
  • Visibility timeout with automatic retry on worker crash
  • DLQ integration for messages exceeding retry limit
  • Thread-safe delivery acknowledgment
  • Moved Store in separate package to avoid circular dependency with mocks

Test Plan

bazel test

@behinddwalls behinddwalls requested review from a team and sbalabanov as code owners February 17, 2026 04:59
@behinddwalls behinddwalls force-pushed the sql-queue-subscriber branch 2 times, most recently from 010ac2e to d0f0ae6 Compare February 19, 2026 04:02
@behinddwalls behinddwalls force-pushed the sql-queue-publisher branch 2 times, most recently from c0eea0f to 2577b76 Compare February 19, 2026 04:28
@behinddwalls behinddwalls force-pushed the sql-queue-publisher branch 2 times, most recently from 3a8459a to c19e90d Compare February 19, 2026 19:00
Comment thread extensions/queue/sql/offset_store.go Outdated
Comment thread extensions/queue/sql/offset_store.go Outdated
Comment thread extensions/queue/sql/offset_store_test.go Outdated
Comment thread extensions/queue/sql/partition_lease_store.go Outdated
Comment thread extensions/queue/sql/partition_lease_store.go Outdated
Comment thread extensions/queue/sql/test_helpers.go Outdated
Comment thread extensions/queue/sql/test_helpers.go Outdated
Comment thread extensions/queue/sql/test_helpers.go Outdated
Comment thread extensions/queue/sql/test_helpers.go Outdated
Comment thread MODULE.bazel Outdated
Comment thread extensions/queue/sql/stores.go Outdated
Comment thread extensions/queue/sql/stores.go Outdated
Comment thread extensions/queue/sql/subscriber.go
Comment thread extensions/queue/sql/subscriber.go Outdated
Comment thread extensions/queue/sql/subscriber.go
Comment thread extensions/queue/sql/subscriber.go
behinddwalls added a commit that referenced this pull request Feb 20, 2026
## Summary
### Why?
Need Publisher implementation to enable message publishing to SQL queue
topics.

### What?

- Publisher validates topic names and publishes single messages via
MessageStore
- Thread-safe with RWMutex for concurrent publish calls
- Idempotent Close() operation
- Comprehensive test coverage for publish, validation, metrics, and
concurrency

- Single and multiple message publishing tested
- Invalid topic names rejected (uppercase, special chars, empty)
- Publisher closed state prevents further publishes
- Concurrent publish operations verified thread-safe
- Context cancellation handled correctly

## Test Plan
make test

## Issues


## Stack
1. @ #22
1. #23
1. #21
1. #24
1. #34
@behinddwalls behinddwalls changed the base branch from sql-queue-publisher to main February 20, 2026 21:29
…cking

- Implement Subscriber interface with partition-based message polling
- Add partition lease management for distributed workers
- Add offset tracking per partition for consumption progress
- Extend MessageStore interface with FetchByOffset, SetVisibilityTimeout, MoveToDLQ
- Add OffsetStore interface for offset management
- Add PartitionLeaseStore interface for partition leasing
- Generate mocks for all three store interfaces
- Add standardized metric tags (topic + partition_key) across all operations
- Add comprehensive test coverage for subscription, ack/nack, and error handling
@behinddwalls behinddwalls merged commit 604a616 into main Feb 20, 2026
1 check passed
@behinddwalls behinddwalls deleted the sql-queue-subscriber branch February 20, 2026 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants